perf(serverless): range-parallel VolumeCache hydrate by deanq · Pull Request #621 · runpod/runpod-python

deanq · 2026-10-14T23:34:24Z

Stacked on #620 (fix/volume-cache-metadata-eperm). Merge that first; this PR then retargets to main.

Bug

Even with #620, VolumeCache.hydrate() on a Global Store volume silently restores only part of a model. Hydrate copies whole big files in parallel with shutil.copyfile, which uses sendfile on Linux. Under that load the geesefs mount fails reads with ENOMEM. _copy_file logs a warning and drops the file, nothing retries it, and the worker falls back to a cold download.

It's also slow when it works. The mount serves one sequential stream at ~140 MiB/s, and reading several whole files at once doesn't add up.

Measured on a 4090 pod in EU-RO-1. A 15.2 GB Qwen2.5-7B was synced to a global volume, then hydrated into an empty dir, alternating implementations, with the page cache dropped between runs:

hydrate	files restored	time
#620	9 of 11 (2 of 4 weight shards lost to `ENOMEM`), both runs	53.5s, 24.2s
this PR	11 of 11, byte-identical to the source (`diff -r`)	21.7s, 17.2s (670–844 MiB/s)

Fix

Hydrate splits every big file into hydrate_chunk_size ranges (default 8 MiB). One pool of hydrate_workers threads (default 64) copies the ranges of all files with pread/pwrite, each range on its own descriptors. Independent range reads scale to the host NIC (~1.1 GiB/s measured), where whole-file copies don't.

Ranges go in offset-major order, so concurrent readers spread across every file instead of queuing on one.
Each range retries ENOMEM/EIO/EAGAIN with exponential backoff (8 attempts, ~6s total). Any other error, or exhausted retries, fails the file, and its remaining ranges are skipped.
A file is assembled in a preallocated temp file and os.replaced into place only after every range has copied, stamped with the manifest's source mtime as in fix(serverless): VolumeCache copies files on volumes that reject utime/chmod #620. A failed file leaves any existing local copy untouched and is not counted as restored.
A short read (a mirror file smaller than its manifest entry) fails the file instead of restoring zero padding. It isn't retried.
hydrate_workers and hydrate_chunk_size are validated like max_workers.
Bytes in flight (workers × chunk) is the knob that matters. The default is 512 MiB, and that's also the peak hydrate memory. In earlier testbed runs, much past ~1 GiB in flight triggered ENOMEM storms on busy hosts.
sync() is unchanged. Writes to the mount scale with the number of parallel files (~400 MiB/s at 8), and range writes into one geesefs object haven't been validated.
Cleanup: dropped the now-unused mtime_of/mtime plumbing from _copy_parallel/_copy_file, extracted a _tmp_path helper (4 call sites), and added _positive_or_default for constructor validation.
docs/serverless/volume_cache.md documents the new options and the hydrate path.

Tests

14 new tests in test_rp_volume_cache.py, written first and confirmed failing:

The range split covers every byte, including a final partial chunk.
Ranges interleave across files.
Transient ENOMEM is retried and the file restored.
A non-transient error is not retried and aborts that file only.
A persistently failing file is skipped while the others restore, with no temp files left.
A failure leaves an existing local file untouched.
A truncated mirror is rejected.
Failures creating the local temp file and failures on rename are handled.
Argument validation and defaults.

make quality-check passes: 1492 tests, 91.7% total coverage, and rp_volume_cache.py is at 97%. The remaining uncovered lines predate this change.

Copilot

🟡 Changes recommended

Concurrent source replacement can corrupt restored files, and scheduling memory and argument validation need correction.

3 open findings

Prevent mixed-generation data across range reads · New
Bound range task creation and outstanding futures · New
Reject non-integer hydrate chunk sizes early · New

What changed in this PR

Builds on #620 to improve VolumeCache hydration speed and reliability on Global Store volumes.

Changes:

Replaces whole-file hydration with configurable parallel range reads and transient-error retries.
Assembles files in temporary storage before atomic publication.
Adds regression tests and documents hydration tuning.

File	Description
tests/test_serverless/test_utils/test_rp_volume_cache.py	Tests range copying, retries, cleanup, and configuration.
runpod/serverless/utils/rp_volume_cache.py	Implements range-parallel hydration and shared helpers.
docs/serverless/volume_cache.md	Documents hydration behavior and tuning options.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Address review feedback on #621: - Pin each mirror file's size+mtime before its first range and reject the file if a range opens a replaced source, or if the mirror size disagrees with the manifest (a sync that landed after the manifest was read), instead of publishing mixed or truncated bytes. - Generate ranges lazily and pull them from hydrate_workers threads so scheduling stays O(workers) instead of one future per range. - Stop remaining ranges and remove pre-sized temp files when an unexpected (non-OSError) error aborts the hydrate. - Reject non-integer and bool tuning values at construction.

marie-karpinska · 2026-10-19T19:05:37Z

+
+    with ThreadPoolExecutor(max_workers=workers) as pool:
+        for future in [pool.submit(drain) for _ in range(workers)]:
+            future.result()


if hydrate gets interrupted here (say Ctrl-C), will the threads keep copying every remaining chunk before it stops? stop only gets set when a thread hits an error. maybe call stop.set() when this loop raises?

marie-karpinska · 2026-10-19T19:06:46Z

+        except OSError as exc:
+            if exc.errno not in _RANGE_RETRY_ERRNOS:
+                raise
+            time.sleep(_RANGE_RETRY_BASE_SECONDS * 2**attempt)


should we add a bit of randomness to the wait? when the mount gets overloaded, all 64 threads fail together and would retry at the same moment, which could overload it again

marie-karpinska · 2026-10-19T19:07:22Z

+        self._max_workers = _positive_or_default(
+            "max_workers", max_workers, min(32, (os.cpu_count() or 4) * 4)
+        )
+        self._hydrate_workers = _positive_or_default("hydrate_workers", hydrate_workers, _HYDRATE_WORKERS)


max_workers doesn't affect hydrate anymore, right? so anyone who set it low to keep hydrate light now gets 64 threads and up to 512 MiB of memory. should we mention this in the changelog, or use max_workers for hydrate when it's set and hydrate_workers isn't?

Hydrate now splits every big file into 8 MiB ranges and copies the ranges of all files from one pool of 64 threads, each range on its own fds. Object-store mounts (Global Store's geesefs) serve one sequential stream at ~140 MiB/s, and whole-file parallelism doesn't add up. Worse, parallel whole-file copies there fail with ENOMEM, and hydrate dropped those files without retry. Measured on a 4090 pod in EU-RO-1, hydrating a 15.2 GB 7B model from a global volume: before: 9 of 11 files (2 of 4 weight shards lost to ENOMEM), 24-53s after: 11 of 11 files, byte-identical, 17-22s (670-844 MiB/s) ENOMEM/EIO/EAGAIN are retried per range with backoff. Files are assembled in a temp file and renamed only when every range copied, so a failure leaves the local file untouched. A mirror shorter than its manifest entry fails instead of restoring zero padding. Tunable via hydrate_workers and hydrate_chunk_size. Sync is unchanged.

Address review feedback on #621: - Pin each mirror file's size+mtime before its first range and reject the file if a range opens a replaced source, or if the mirror size disagrees with the manifest (a sync that landed after the manifest was read), instead of publishing mixed or truncated bytes. - Generate ranges lazily and pull them from hydrate_workers threads so scheduling stays O(workers) instead of one future per range. - Stop remaining ranges and remove pre-sized temp files when an unexpected (non-OSError) error aborts the hydrate. - Reject non-integer and bool tuning values at construction.

Hoist the last attempt out of the retry loop so _with_range_retry has no implicit fall-through return (CodeQL py/mixed-returns). Adds a test pinning the attempt count when transient errors persist.

_with_range_retry only wraps _copy_range, which returns None, so stop forwarding its result (CodeQL py/procedure-return-value-used).

deanq requested a balanced review from Copilot October 14, 2026 23:47

Copilot started reviewing on behalf of deanq October 14, 2026 23:48

View session

Copilot AI reviewed Oct 14, 2026

View reviewed changes

Comment thread runpod/serverless/utils/rp_volume_cache.py

Comment thread runpod/serverless/utils/rp_volume_cache.py Outdated

Comment thread runpod/serverless/utils/rp_volume_cache.py Outdated

deanq requested a review from a team October 15, 2026 01:24

Base automatically changed from fix/volume-cache-metadata-eperm to main October 15, 2026 06:07

deanq force-pushed the feature/volume-cache-parallel-hydrate branch from 0037705 to 51ffc0a Compare October 15, 2026 07:14

github-code-quality Bot found potential problems Oct 15, 2026

View reviewed changes

Comment thread runpod/serverless/utils/rp_volume_cache.py Fixed

github-advanced-security Bot found potential problems Oct 15, 2026

View reviewed changes

Comment thread runpod/serverless/utils/rp_volume_cache.py Fixed

github-advanced-security Bot found potential problems Oct 16, 2026

View reviewed changes

Comment thread runpod/serverless/utils/rp_volume_cache.py Fixed

Comment thread runpod/serverless/utils/rp_volume_cache.py Fixed

github-code-quality Bot found potential problems Oct 16, 2026

View reviewed changes

Comment thread runpod/serverless/utils/rp_volume_cache.py Fixed

Comment thread runpod/serverless/utils/rp_volume_cache.py Fixed

deanq requested a review from justinwlin October 16, 2026 00:56

deanq force-pushed the feature/volume-cache-parallel-hydrate branch from 92fe352 to 63f9905 Compare October 19, 2026 17:21

deanq force-pushed the feature/volume-cache-parallel-hydrate branch from 63f9905 to b2cbacb Compare October 19, 2026 17:47

marie-karpinska reviewed Oct 19, 2026

View reviewed changes

deanq and others added 4 commits October 20, 2026 16:45

refactor(serverless): make range retry's final attempt explicit

dfa5684

Hoist the last attempt out of the retry loop so _with_range_retry has no implicit fall-through return (CodeQL py/mixed-returns). Adds a test pinning the attempt count when transient errors persist.

refactor(serverless): treat range retry as a procedure

de1576d

_with_range_retry only wraps _copy_range, which returns None, so stop forwarding its result (CodeQL py/procedure-return-value-used).

deanq force-pushed the feature/volume-cache-parallel-hydrate branch from b2cbacb to de1576d Compare October 20, 2026 23:47

Navigation Menu

Repository navigation

perf(serverless): range-parallel VolumeCache hydrate - #621

perf(serverless): range-parallel VolumeCache hydrate#621
deanq wants to merge 4 commits into
mainfrom
feature/volume-cache-parallel-hydrate

deanq commented Oct 14, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

marie-karpinska Oct 19, 2026

Uh oh!

marie-karpinska Oct 19, 2026

Uh oh!

marie-karpinska Oct 19, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

deanq commented Oct 14, 2026

Bug

Fix

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

marie-karpinska Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

marie-karpinska Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

marie-karpinska Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants